Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[fix](serde)fix the bug in DataTypeNullableSerDe.deserialize_column_from_fixed_json #38245

Merged
merged 3 commits into from
Aug 2, 2024
Merged

[fix](serde)fix the bug in DataTypeNullableSerDe.deserialize_column_from_fixed_json #38245

merged 3 commits into from
Aug 2, 2024

Conversation

hubgeter
Copy link
Contributor

@hubgeter hubgeter commented Jul 23, 2024 •
edited
Loading

Proposed changes

fix a bug in DataTypeNullableSerDe.deserialize_column_from_fixed_json.

The expected behavior of the deserialize_column_from_fixed_json function is to insert n values ​​into the column.

However, when the DataTypeNullableSerDe class implements this function, the null_map column is resize to n, which does not insert n values ​​into it. Since this function is only used by the _fill_partition_columns of the parquet/orc reader and is not called repeatedly for a get_next_block, this bug is covered up.
before pr : #37377

@doris-robot
Copy link

Thank you for your contribution to Apache Doris.
Don't know what should be done next? See How to process your PR

Since 2024-03-18, the Document has been moved to doris-website.
See Doris Document.

@hubgeter
Copy link
Contributor Author

run buildall

@github-actions GitHub Actions
Copy link
Contributor

clang-tidy review says "All clean, LGTM! 👍"

@doris-robot
Copy link

TPC-H: Total hot run time: 39840 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit b97b34cd3e86265d2d40faf8e9958c402b899344, data reload: false

------ Round 1 ----------------------------------
q1	17638	4411	4318	4318
q2	2010	193	191	191
q3	10435	1256	1109	1109
q4	10179	749	830	749
q5	7551	2722	2662	2662
q6	222	138	137	137
q7	956	601	586	586
q8	9206	2076	2081	2076
q9	8766	6582	6583	6582
q10	8817	3743	3784	3743
q11	466	243	237	237
q12	477	223	224	223
q13	17834	3009	2981	2981
q14	273	231	232	231
q15	528	496	501	496
q16	500	380	383	380
q17	981	658	610	610
q18	8190	7493	7493	7493
q19	6067	1401	1215	1215
q20	689	321	323	321
q21	5120	3213	3234	3213
q22	351	294	287	287
Total cold run time: 117256 ms
Total hot run time: 39840 ms

----- Round 2, with runtime_filter_mode=off -----
q1	4367	4378	4328	4328
q2	375	275	270	270
q3	3001	2891	2889	2889
q4	1987	1721	1701	1701
q5	5664	5524	5492	5492
q6	228	135	133	133
q7	2219	1878	1865	1865
q8	3267	3396	3453	3396
q9	8852	8807	8914	8807
q10	4121	3935	3791	3791
q11	606	496	500	496
q12	834	648	657	648
q13	15822	3186	3190	3186
q14	312	294	285	285
q15	542	511	507	507
q16	482	450	430	430
q17	1820	1555	1520	1520
q18	8263	8115	7879	7879
q19	2506	1628	1454	1454
q20	2145	1898	1898	1898
q21	5181	4977	4950	4950
q22	560	488	497	488
Total cold run time: 73154 ms
Total hot run time: 56413 ms

@doris-robot
Copy link

TPC-DS: Total hot run time: 173945 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit b97b34cd3e86265d2d40faf8e9958c402b899344, data reload: false

query1	902	371	369	369
query2	6435	1913	1814	1814
query3	6636	218	214	214
query4	24812	17552	17574	17552
query5	3635	475	494	475
query6	285	178	170	170
query7	4582	290	285	285
query8	246	206	198	198
query9	8659	2412	2378	2378
query10	435	289	284	284
query11	11642	10065	10181	10065
query12	115	89	90	89
query13	1638	376	367	367
query14	10328	7779	7915	7779
query15	214	166	171	166
query16	7756	467	466	466
query17	1543	549	528	528
query18	1960	289	267	267
query19	202	147	148	147
query20	89	81	84	81
query21	208	129	124	124
query22	4405	4014	3946	3946
query23	34127	34015	33780	33780
query24	10956	3062	2966	2966
query25	627	389	400	389
query26	726	152	153	152
query27	2296	277	287	277
query28	5902	2033	2030	2030
query29	864	635	652	635
query30	269	160	160	160
query31	971	771	784	771
query32	99	59	56	56
query33	657	358	364	358
query34	929	518	506	506
query35	882	774	765	765
query36	1177	972	963	963
query37	137	87	86	86
query38	2988	2809	2790	2790
query39	883	813	832	813
query40	213	128	126	126
query41	49	46	48	46
query42	120	104	98	98
query43	499	463	482	463
query44	1100	725	723	723
query45	190	163	164	163
query46	1107	727	716	716
query47	1846	1802	1785	1785
query48	381	305	289	289
query49	829	417	424	417
query50	772	385	392	385
query51	6725	6741	6670	6670
query52	102	96	97	96
query53	359	291	292	291
query54	879	452	447	447
query55	76	73	73	73
query56	299	274	273	273
query57	1133	1074	1035	1035
query58	256	267	248	248
query59	2823	2727	2433	2433
query60	301	275	281	275
query61	98	97	97	97
query62	800	674	653	653
query63	335	310	296	296
query64	9149	2210	1692	1692
query65	3123	3096	3107	3096
query66	759	334	334	334
query67	15534	14861	15092	14861
query68	4666	528	540	528
query69	674	449	384	384
query70	1212	1153	1102	1102
query71	454	289	296	289
query72	7276	5085	5602	5085
query73	752	319	321	319
query74	6161	5721	5686	5686
query75	3598	2655	2706	2655
query76	3306	911	918	911
query77	612	314	315	314
query78	9578	9098	9208	9098
query79	1499	518	540	518
query80	2463	542	497	497
query81	611	224	225	224
query82	595	144	137	137
query83	318	167	167	167
query84	262	84	87	84
query85	1110	328	312	312
query86	461	319	309	309
query87	3298	3093	3144	3093
query88	3718	2357	2381	2357
query89	480	397	399	397
query90	1728	201	200	200
query91	139	102	100	100
query92	64	51	50	50
query93	1995	488	491	488
query94	1156	301	298	298
query95	418	330	325	325
query96	599	272	280	272
query97	3177	2975	3047	2975
query98	208	201	200	200
query99	1496	1229	1271	1229
Total cold run time: 275158 ms
Total hot run time: 173945 ms

@doris-robot
Copy link

ClickBench: Total hot run time: 30.29 s 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit b97b34cd3e86265d2d40faf8e9958c402b899344, data reload: false

query1	0.04	0.04	0.04
query2	0.08	0.04	0.04
query3	0.23	0.05	0.04
query4	1.67	0.08	0.08
query5	0.50	0.48	0.48
query6	1.13	0.72	0.72
query7	0.02	0.01	0.01
query8	0.05	0.04	0.04
query9	0.56	0.50	0.49
query10	0.54	0.55	0.54
query11	0.15	0.12	0.11
query12	0.15	0.13	0.12
query13	0.60	0.60	0.58
query14	0.75	0.79	0.80
query15	0.84	0.80	0.81
query16	0.35	0.36	0.37
query17	0.96	0.95	0.96
query18	0.23	0.24	0.21
query19	1.77	1.63	1.68
query20	0.02	0.01	0.01
query21	15.38	0.81	0.65
query22	4.25	7.76	1.58
query23	18.27	1.38	1.36
query24	2.07	0.24	0.22
query25	0.15	0.09	0.08
query26	0.30	0.21	0.21
query27	0.46	0.23	0.23
query28	13.30	1.00	0.99
query29	12.59	3.28	3.30
query30	0.25	0.06	0.05
query31	2.84	0.40	0.40
query32	3.26	0.48	0.46
query33	2.90	2.88	2.98
query34	17.02	4.34	4.43
query35	4.52	4.46	4.49
query36	0.65	0.49	0.49
query37	0.19	0.16	0.15
query38	0.15	0.15	0.15
query39	0.04	0.04	0.03
query40	0.15	0.12	0.12
query41	0.10	0.05	0.04
query42	0.05	0.04	0.05
query43	0.04	0.04	0.04
Total cold run time: 109.57 s
Total hot run time: 30.29 s

@hubgeter hubgeter force-pushed the fix_nullable_serde_fixed_value branch from b97b34c to 8903bcf Compare July 29, 2024 03:55
@github-actions GitHub Actions
Copy link
Contributor

clang-tidy review says "All clean, LGTM! 👍"

@hubgeter
Copy link
Contributor Author

run buildall

@doris-robot
Copy link

TPC-H: Total hot run time: 41830 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 8903bcf24367f25ba9853db4ed5788adea9231bc, data reload: false

------ Round 1 ----------------------------------
q1	20543	4998	4073	4073
q2	2016	212	200	200
q3	10495	1364	1350	1350
q4	10164	825	963	825
q5	7593	2979	2983	2979
q6	224	136	137	136
q7	1058	616	615	615
q8	9437	1932	1948	1932
q9	8491	6597	6610	6597
q10	8686	3842	3829	3829
q11	422	255	256	255
q12	402	230	231	230
q13	17740	2942	2955	2942
q14	266	240	247	240
q15	532	482	492	482
q16	485	414	386	386
q17	972	930	925	925
q18	7981	7279	7294	7279
q19	1371	1218	1202	1202
q20	565	321	337	321
q21	5406	4746	4781	4746
q22	356	292	286	286
Total cold run time: 115205 ms
Total hot run time: 41830 ms

----- Round 2, with runtime_filter_mode=off -----
q1	4063	4033	4042	4033
q2	332	226	232	226
q3	2986	3027	2980	2980
q4	1866	1862	1868	1862
q5	5231	5247	5245	5245
q6	221	133	131	131
q7	2040	1671	1764	1671
q8	3198	3311	3289	3289
q9	8345	8278	8303	8278
q10	3771	3853	3812	3812
q11	544	455	451	451
q12	749	590	572	572
q13	13073	2957	2969	2957
q14	301	258	255	255
q15	519	483	483	483
q16	448	407	402	402
q17	1738	1691	1698	1691
q18	7728	7244	7191	7191
q19	1668	1671	1663	1663
q20	1982	1779	1782	1779
q21	5557	5251	5286	5251
q22	533	452	483	452
Total cold run time: 66893 ms
Total hot run time: 54674 ms

@doris-robot
Copy link

TPC-DS: Total hot run time: 169221 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 8903bcf24367f25ba9853db4ed5788adea9231bc, data reload: false

query1	920	371	364	364
query2	6482	1640	1582	1582
query3	6686	215	221	215
query4	19688	17665	17313	17313
query5	4269	509	505	505
query6	286	172	161	161
query7	4596	318	295	295
query8	267	194	195	194
query9	8526	2394	2390	2390
query10	436	286	265	265
query11	10487	10056	10089	10056
query12	143	90	87	87
query13	1642	387	390	387
query14	9469	6996	8589	6996
query15	201	171	171	171
query16	7078	473	460	460
query17	965	590	587	587
query18	1923	296	291	291
query19	196	152	150	150
query20	91	89	86	86
query21	206	105	104	104
query22	4222	3983	3902	3902
query23	33672	33041	32961	32961
query24	10441	3144	3037	3037
query25	718	397	403	397
query26	1787	150	152	150
query27	3028	278	282	278
query28	7015	2010	1994	1994
query29	1387	439	445	439
query30	287	148	150	148
query31	925	796	762	762
query32	104	55	63	55
query33	722	319	352	319
query34	900	487	492	487
query35	823	750	749	749
query36	1005	866	872	866
query37	298	80	79	79
query38	2861	2817	2764	2764
query39	905	831	833	831
query40	282	117	115	115
query41	50	46	47	46
query42	128	100	106	100
query43	474	428	423	423
query44	1202	740	731	731
query45	205	181	187	181
query46	1086	801	782	782
query47	1812	1670	1705	1670
query48	366	305	297	297
query49	1215	442	423	423
query50	895	439	439	439
query51	6845	6788	6700	6700
query52	104	95	91	91
query53	256	182	180	180
query54	647	465	489	465
query55	80	79	79	79
query56	288	268	280	268
query57	1143	1044	1021	1021
query58	281	282	284	282
query59	2501	2315	2389	2315
query60	301	297	309	297
query61	121	213	102	102
query62	913	681	676	676
query63	215	187	178	178
query64	5908	1930	1920	1920
query65	3154	3112	3104	3104
query66	1435	340	338	338
query67	15513	14760	14750	14750
query68	9145	585	597	585
query69	738	389	316	316
query70	1178	1045	1040	1040
query71	519	283	282	282
query72	7963	2716	2492	2492
query73	1700	343	335	335
query74	6066	5693	5606	5606
query75	4545	2757	2746	2746
query76	5775	1393	1421	1393
query77	736	318	302	302
query78	9640	8934	8864	8864
query79	6933	544	536	536
query80	1010	530	539	530
query81	545	227	234	227
query82	773	129	128	128
query83	234	181	175	175
query84	272	80	84	80
query85	1429	334	304	304
query86	419	278	306	278
query87	3270	3122	3094	3094
query88	4000	2463	2476	2463
query89	447	286	290	286
query90	1961	190	188	188
query91	125	103	101	101
query92	61	49	49	49
query93	5537	616	619	616
query94	862	295	269	269
query95	395	274	275	274
query96	606	289	284	284
query97	3203	3040	3041	3040
query98	215	197	194	194
query99	1640	1294	1302	1294
Total cold run time: 285019 ms
Total hot run time: 169221 ms

@doris-robot
Copy link

ClickBench: Total hot run time: 30 s 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 8903bcf24367f25ba9853db4ed5788adea9231bc, data reload: false

query1	0.04	0.04	0.03
query2	0.07	0.03	0.04
query3	0.22	0.04	0.04
query4	1.69	0.06	0.07
query5	0.48	0.49	0.48
query6	1.14	0.73	0.71
query7	0.02	0.02	0.01
query8	0.05	0.04	0.04
query9	0.56	0.51	0.52
query10	0.56	0.57	0.57
query11	0.15	0.12	0.12
query12	0.15	0.13	0.13
query13	0.61	0.60	0.61
query14	0.77	0.79	0.79
query15	0.89	0.86	0.88
query16	0.36	0.35	0.36
query17	1.00	1.03	1.04
query18	0.23	0.21	0.21
query19	1.80	1.83	1.74
query20	0.02	0.03	0.01
query21	15.42	0.78	0.67
query22	4.02	7.34	1.26
query23	17.73	1.29	1.25
query24	2.25	0.23	0.22
query25	0.18	0.08	0.08
query26	0.31	0.22	0.21
query27	0.47	0.24	0.24
query28	13.17	0.99	0.96
query29	12.51	3.36	3.31
query30	0.26	0.05	0.05
query31	2.85	0.42	0.40
query32	3.24	0.50	0.49
query33	2.90	2.99	2.94
query34	15.47	4.27	4.25
query35	4.31	4.32	4.29
query36	0.66	0.49	0.48
query37	0.18	0.15	0.18
query38	0.15	0.16	0.15
query39	0.04	0.03	0.04
query40	0.16	0.13	0.14
query41	0.10	0.04	0.05
query42	0.06	0.04	0.05
query43	0.04	0.05	0.04
Total cold run time: 107.29 s
Total hot run time: 30 s

morningman
morningman previously approved these changes Jul 30, 2024
View reviewed changes
Copy link
Contributor

@morningman morningman left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@github-actions GitHub Actions
Copy link
Contributor

PR approved by at least one committer and no changes requested.

@github-actions github-actions bot added the approved Indicates a PR has been approved by one committer. label Jul 30, 2024
@github-actions GitHub Actions
Copy link
Contributor

PR approved by anyone and no changes requested.

@hubgeter
Copy link
Contributor Author

run buildall

@github-actions github-actions bot removed the approved Indicates a PR has been approved by one committer. label Jul 31, 2024
@github-actions GitHub Actions
Copy link
Contributor

clang-tidy review says "All clean, LGTM! 👍"

@doris-robot
Copy link

TPC-H: Total hot run time: 42288 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit ecb56acb20f099434d990324d57f581112293d6b, data reload: false

------ Round 1 ----------------------------------
q1	17643	4246	4132	4132
q2	2035	216	206	206
q3	10444	1313	1385	1313
q4	10157	870	929	870
q5	7644	3032	3061	3032
q6	229	141	143	141
q7	1067	618	618	618
q8	9454	1997	1960	1960
q9	8602	6692	6664	6664
q10	8752	3868	3894	3868
q11	427	264	274	264
q12	440	242	239	239
q13	17776	2963	2933	2933
q14	276	248	244	244
q15	532	487	490	487
q16	502	396	390	390
q17	1005	953	934	934
q18	8161	7407	7349	7349
q19	1377	1234	1239	1234
q20	580	340	353	340
q21	5533	4782	4890	4782
q22	356	288	293	288
Total cold run time: 112992 ms
Total hot run time: 42288 ms

----- Round 2, with runtime_filter_mode=off -----
q1	4091	4057	4057	4057
q2	332	225	220	220
q3	3021	3139	3193	3139
q4	2086	2031	1996	1996
q5	5686	5545	5537	5537
q6	225	136	136	136
q7	2152	1822	1816	1816
q8	3374	3456	3394	3394
q9	8782	8742	8868	8742
q10	3961	4075	4019	4019
q11	547	469	463	463
q12	781	641	660	641
q13	16205	3136	3160	3136
q14	305	267	283	267
q15	540	495	499	495
q16	452	427	422	422
q17	1781	1782	1764	1764
q18	8191	7942	7767	7767
q19	1776	1741	1768	1741
q20	2085	1868	1850	1850
q21	5795	5577	5415	5415
q22	537	471	476	471
Total cold run time: 72705 ms
Total hot run time: 57488 ms

@doris-robot
Copy link

TPC-DS: Total hot run time: 171261 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit ecb56acb20f099434d990324d57f581112293d6b, data reload: false

query1	912	381	397	381
query2	6473	1725	1676	1676
query3	6670	225	226	225
query4	20460	17516	17427	17427
query5	3663	528	514	514
query6	274	196	174	174
query7	4592	306	293	293
query8	253	189	196	189
query9	8514	2409	2405	2405
query10	423	265	267	265
query11	10612	10201	10032	10032
query12	120	86	90	86
query13	1637	384	376	376
query14	8921	7880	7708	7708
query15	208	167	168	167
query16	6815	503	464	464
query17	958	556	567	556
query18	1932	298	294	294
query19	205	152	149	149
query20	92	88	86	86
query21	202	104	102	102
query22	4202	4130	3974	3974
query23	34268	33774	33652	33652
query24	10214	3115	3084	3084
query25	687	435	429	429
query26	1780	158	148	148
query27	2954	293	301	293
query28	7598	2049	2026	2026
query29	1369	460	430	430
query30	232	150	152	150
query31	970	784	770	770
query32	105	56	57	56
query33	673	334	342	334
query34	965	515	506	506
query35	883	773	767	767
query36	1079	885	892	885
query37	299	81	84	81
query38	2938	2854	2796	2796
query39	871	810	824	810
query40	250	114	112	112
query41	43	49	44	44
query42	123	98	100	98
query43	464	413	441	413
query44	1218	737	741	737
query45	208	178	180	178
query46	1122	820	821	820
query47	1799	1718	1711	1711
query48	371	300	289	289
query49	954	438	429	429
query50	937	436	438	436
query51	6792	6656	6620	6620
query52	110	90	90	90
query53	262	186	186	186
query54	616	452	468	452
query55	77	73	77	73
query56	270	258	262	258
query57	1130	1034	1071	1034
query58	262	293	308	293
query59	2706	2495	2607	2495
query60	303	278	291	278
query61	97	94	97	94
query62	909	672	653	653
query63	207	182	185	182
query64	5710	1904	1899	1899
query65	3175	3142	3091	3091
query66	1319	356	327	327
query67	15270	14920	14887	14887
query68	4351	555	582	555
query69	453	297	295	295
query70	1085	1072	1062	1062
query71	453	287	278	278
query72	7190	2658	2542	2542
query73	796	331	331	331
query74	6090	5619	5623	5619
query75	3434	2754	2730	2730
query76	2237	1355	1397	1355
query77	435	308	303	303
query78	9473	8900	8960	8900
query79	1983	535	548	535
query80	1139	510	508	508
query81	555	228	224	224
query82	1052	131	127	127
query83	250	182	171	171
query84	268	80	81	80
query85	1250	316	313	313
query86	392	287	276	276
query87	3292	3122	3097	3097
query88	2995	2486	2397	2397
query89	387	297	290	290
query90	1719	195	192	192
query91	130	101	101	101
query92	60	50	51	50
query93	1535	631	618	618
query94	814	298	302	298
query95	390	280	275	275
query96	597	279	283	279
query97	3232	3107	3039	3039
query98	215	202	193	193
query99	1604	1289	1294	1289
Total cold run time: 262583 ms
Total hot run time: 171261 ms

@doris-robot
Copy link

ClickBench: Total hot run time: 30.38 s 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit ecb56acb20f099434d990324d57f581112293d6b, data reload: false

query1	0.04	0.04	0.04
query2	0.07	0.04	0.03
query3	0.23	0.04	0.05
query4	1.69	0.07	0.06
query5	0.50	0.48	0.49
query6	1.14	0.72	0.71
query7	0.02	0.02	0.02
query8	0.06	0.04	0.04
query9	0.58	0.52	0.52
query10	0.58	0.58	0.56
query11	0.16	0.12	0.11
query12	0.15	0.12	0.13
query13	0.61	0.61	0.60
query14	0.77	0.78	0.81
query15	0.90	0.87	0.86
query16	0.35	0.36	0.35
query17	1.02	0.98	1.00
query18	0.22	0.21	0.22
query19	1.92	1.76	1.78
query20	0.01	0.00	0.00
query21	15.48	0.79	0.68
query22	3.94	6.55	1.54
query23	17.79	1.43	1.30
query24	2.33	0.23	0.21
query25	0.17	0.09	0.08
query26	0.32	0.21	0.21
query27	0.46	0.24	0.23
query28	13.16	1.00	0.98
query29	12.61	3.39	3.31
query30	0.27	0.06	0.05
query31	2.86	0.41	0.41
query32	3.24	0.49	0.50
query33	3.00	2.95	2.95
query34	15.46	4.32	4.29
query35	4.31	4.30	4.32
query36	0.67	0.48	0.49
query37	0.18	0.16	0.16
query38	0.16	0.15	0.15
query39	0.04	0.04	0.03
query40	0.15	0.13	0.13
query41	0.10	0.05	0.05
query42	0.05	0.04	0.05
query43	0.04	0.05	0.04
Total cold run time: 107.81 s
Total hot run time: 30.38 s

morningman
morningman approved these changes Aug 1, 2024
View reviewed changes
Copy link
Contributor

@morningman morningman left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@github-actions GitHub Actions
Copy link
Contributor

github-actions bot commented Aug 1, 2024

PR approved by at least one committer and no changes requested.

@github-actions github-actions bot added the approved Indicates a PR has been approved by one committer. label Aug 1, 2024
kaka11chen
kaka11chen approved these changes Aug 1, 2024
View reviewed changes
Copy link
Contributor

@kaka11chen kaka11chen left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@morningman morningman merged commit 5c52581 into apache:master Aug 2, 2024
27 of 31 checks passed
hubgeter added a commit to hubgeter/doris that referenced this pull request Aug 2, 2024
@hubgeter
[fix](serde)fix the bug in DataTypeNullableSerDe.deserialize_column_f…
2b7b903 
…rom_fixed_json (apache#38245)

## Proposed changes
fix a bug in DataTypeNullableSerDe.deserialize_column_from_fixed_json.

The expected behavior of the `deserialize_column_from_fixed_json`
function is to `insert` n values ​​into the column.

However, when the `DataTypeNullableSerDe` class implements this
function, the null_map column is `resize` to n, which does not insert n
values ​​into it. Since this function is only used by the
`_fill_partition_columns` of the `parquet/orc reader` and is not called
repeatedly for a `get_next_block`, this bug is covered up.
before pr : apache#37377
@hubgeter hubgeter mentioned this pull request Aug 2, 2024
Merged
yiguolei pushed a commit that referenced this pull request Aug 5, 2024
@hubgeter
[opt](serde)Optimize the filling of fixed values ​​into block columns…
607c0b8 
… without repeated deserialization. (#37377) (#38245) (#38810)

## Proposed changes
pick pr: #38575  and fix this pr bug :  #38245
dataroaring pushed a commit that referenced this pull request Aug 11, 2024
@hubgeter @dataroaring
[fix](serde)fix the bug in DataTypeNullableSerDe.deserialize_column_f…
eae2cd8 
…rom_fixed_json (#38245)

## Proposed changes
fix a bug in DataTypeNullableSerDe.deserialize_column_from_fixed_json.

The expected behavior of the `deserialize_column_from_fixed_json`
function is to `insert` n values ​​into the column.

However, when the `DataTypeNullableSerDe` class implements this
function, the null_map column is `resize` to n, which does not insert n
values ​​into it. Since this function is only used by the
`_fill_partition_columns` of the `parquet/orc reader` and is not called
repeatedly for a `get_next_block`, this bug is covered up.
before pr : #37377
dataroaring pushed a commit that referenced this pull request Aug 16, 2024
@hubgeter @dataroaring
[fix](serde)fix the bug in DataTypeNullableSerDe.deserialize_column_f…
1ce5b99 
…rom_fixed_json (#38245)

## Proposed changes
fix a bug in DataTypeNullableSerDe.deserialize_column_from_fixed_json.

The expected behavior of the `deserialize_column_from_fixed_json`
function is to `insert` n values ​​into the column.

However, when the `DataTypeNullableSerDe` class implements this
function, the null_map column is `resize` to n, which does not insert n
values ​​into it. Since this function is only used by the
`_fill_partition_columns` of the `parquet/orc reader` and is not called
repeatedly for a `get_next_block`, this bug is covered up.
before pr : #37377
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@kaka11chen kaka11chen kaka11chen approved these changes

@morningman morningman morningman approved these changes

Assignees
No one assigned
Labels
approved Indicates a PR has been approved by one committer. dev/2.1.6-merged dev/3.0.2-merged reviewed
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
6 participants
@hubgeter @doris-robot @kaka11chen @morningman @yiguolei @dataroaring